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FOREWORD 


As  the  U.S.  Army  grapples  with  the  challenge  of  meeting  complex  training  requirements 
with  increasingly  constrained  resources,  Army  trainers  are  exploring  emerging  technologies  and 
innovative  techniques  that  can  be  used  to  achieve  training  goals.  Part  of  this  exploration  has 
been  accomplished  through  the  Force  XXI  Training  Program  (FXXITP).  Established  in  1995  by 
the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  the  FXXITP  is  overseen  by  the 
Directorate  of  Training  and  Doctrine  Development  (DTDD)  of  the  U.S.  Army  Armor  Center 
(USAARMC)  at  Fort  Knox,  Kentucky.  With  the  active  participation  of  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  DTDD  has  sponsored  the  development  of 
numerous  training  products  for  the  program.  The  majority  of  the  work  was  performed  at  the  ARI 
Armored  Forces  Research  Unit  at  Fort  Knox,  with  additional  work  completed  at  the  Infantry 
Forces  Research  Unit  at  Fort  Benning,  Georgia.  The  work  centered  on  the  development  of 
simulation-based,  structured  training  support  packages  (TSPs)  designed  for  the  collective 
training  of  reserve  component  (RC)  and  active  component  (AC)  battalion  and  brigade  staffs. 

As  the  initial  FXXITP  products  began  to  receive  attention  from  potential  users,  the 
TRADOC  Deputy  Chief  of  Staff  for  Training  (DCST)  directed  that  these  products  be  assessed 
while  being  used  by  AC  brigades.  The  assessment  was  to  focus  on  the  utility  of  the  TSPs  in 
supporting  the  brigades’  yearly  training,  specifically  in  preparation  for  National  Training  Center 
rotations.  The  DCST  asked  ARI  to  provide  the  support  for  assessment,  and  this  was 
accomplished  by  means  of  a  project  entitled  Implementation  and  Support  for  the  Assessment  of 
Force  XXI  Training  Program  (ISAT).  The  ARI  conducted  the  assessment  with  the  assistance  of 
four  AC  brigades  between  April  1998  and  May  1999. 

While  the  primary  focus  of  the  DCST  directive  was  on  assessment  of  FXXITP  product 
utility,  ARI  also  supported  the  implementation  of  the  products,  to  ensure  that  the  assessment 
would  take  place  in  a  context  of  full  product  use.  Therefore,  the  assessment  strategy  included 
plans  for  a  support  infrastructure  that  would  facilitate  integration  of  the  FXXITP  products  into 
unit  training  plans. 

The  scope  of  the  assessment,  as  originally  planned,  was  adjusted  continually  over  the  course 
of  the  project  due  to  external  constraints  and  other  demands  on  the  activities  of  the  participating 
units.  As  a  result,  the  assessment  questions  could  not  be  completely  addressed.  This  report 
presents  a  succinct  summary  of  the  conditions  of  the  implementation  and  the  assessment  lessons 
learned  and  recommendations.  A  larger  report.  Operational  Assessment  of  Force  XXI  Training 
Products:  Lessons  for  Successful  Fielding  (Pratt  et  al.,  2000)  discusses  the  background  of  the 
ISAT  project  and  documents  project  activities  and  outcomes  in  more  detail.  The  implementation 
and  support  history,  assessment  results,  and  lessons  learned  were  briefed  to  the  office  of  the 
DCST  on  21  July  1999.  The  findings  summarized  in  this  report  should  support  the  development 
and  fielding  of  TSPs  that  will  improve  the  near-term  readiness  of  the  Army’s  AC  and  RC  forces. 


ZITA  M.  SIMUTIS 
Technical  Director 
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SUMMARY  OF  RECOMMENDATIONS  FOR  SUCCESSFUL  FIELDING 
OF  FORCE  XXI  TRAINING  PRODUCTS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  facing  the  challenge  of  meeting  complex  training  requirements  with 
increasingly  constrained  resources.  As  a  result.  Army  trainers  are  exploring  emerging 
technologies  and  innovative  techniques  that  can  be  used  to  achieve  training  goals.  Much  of  the 
innovation  has  been  accomplished  through  the  Force  XXI  Training  Program  (FXXITP). 

The  FXXITP  was  established  in  1995  by  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  and  is  overseen  by  the  Directorate  of  Training  and  Doctrine  Development  (DTDD) 
of  the  U.S.  Army  Armor  Center  (USAARMC)  at  Fort  Knox,  Kentucky.  The  DTDD  has  worked 
with  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  research 
and  develop  numerous  training  products  for  the  program.  In  1998,  the  TRADOC  Deputy  Chief 
of  Staff  for  Training  (DCST)  directed  that  these  products  be  assessed  while  being  used  by  active 
component  brigades.  While  the  primary  focus  of  the  DCST  directive  was  on  assessment  of 
FXXITP  product  utility,  ARI  directed  that  the  contractor  team  also  support  implementation  of 
the  products,  to  ensure  that  the  assessment  would  take  place  in  a  context  of  full  product  use. 
Therefore,  the  assessment  strategy  included  plans  for  a  support  infrastructure  that  would 
facilitate  integration  of  the  FXXITP  products  into  unit  training  plans. 

The  project  that  was  organized  to  support  the  DCST  directive  was  titled  Implementation  and 
Support  for  the  Assessment  of  Force  XXI  Training  Program  (ISAT).  The  specific  objectives 
included:  (a)  to  develop  a  plan  to  guide  the  implementation  and  assessment  of  selected 
components  of  the  FXXITP;  (b)  to  build  an  implementation  support  infrastructure  at 
participating  unit  locations  to  facilitate  the  assessment;  (c)  to  conduct  the  implementation  and 
assessment  at  unit  locations  and  Fort  Irwin,  California,  (the  National  Training  Center  [NTC]); 
and  (d)  to  document  the  implementation  and  assessment  process,  the  results  of  the  assessment, 
required  changes  to  the  products,  and  suggestions  for  future  development. 

Procedure: 

The  project’s  Execution  Plan  (Human  Resources  Research  Organization  [HumRRO], 
Raytheon  Systems  Company,  TRW  S&ITG,  &  Litton  PRC,  1998b)  outlined  the  project’s 
strategy  for  accomplishing  the  objectives.  It  included  plans  for  managing  the  project  from  Fort 
Knox,  and  providing  support  to  the  two  selected  user  units.  Data  collection  for  the  assessment 
would  involve  interviews,  questionnaires,  and  observations,  as  outlined  in  the  Assessment  Plan 
(HumRRO  et  al.,  1998a). 

The  activities  supporting  the  implementation  included  purchase  and  installation  of 
training  computers  and  associated  components;  reproduction  and  distribution  of  computer-based 
training  programs  for  individual  staff  members  and  training  support  packages  (TSPs)  for  small 
group  and  complex  group  exercises;  hiring  and  training  of  on-site  personnel  at  the  user  unit  sites 
who  would  provide  train-the-trainer  sessions  on  all  FXXITP  products  and  assist  with 
implementation  of  the  individual  computer-based  instruction  and  small  group  exercises;  and  a 
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system  whereby  personnel  from  Fort  Benning,  and  Fort  Knox  could  be  available  to  answer 
difficult  questions  about  the  FXXITP  products  and  could  travel  to  the  user  units  and  assist  with 
implementation  of  the  complex  group  exercises. 

The  assessment  requirement  (HumRRO  et  al.,  1998a)  included  observation  and 
documentation  of  the  implementation  and  assessment  processes;  data  collection  from  participants 
by  means  of  interviews  and  questionnaires;  quantitative  and  qualitative  analyses  of  the 
assessment  data;  and  reporting  of  findings,  lessons  learned,  and  implications  for  FXXITP  future 
directions. 

Because  only  a  limited  number  of  brigades  were  going  to  participate  in  the  implementation 
and  assessment,  and  because  there  was  no  opportunity  for  rigorous  control  of  the  brigade 
activities,  the  assessment  was  not  meant  to  be  a  full  evaluation  of  the  products’  value  or  impact. 
Instead,  ARI  and  DTDD  asked  for  a  complete  case  study  of  the  implementation  activities,  along 
with  intense  efforts  to  obtain  reactions  and  suggestions  from  all  those  involved  in  the  product 
use. 

Findings: 

The  ISAT  project  outcomes  represent  a  compilation  of  implementation  methods,  assessment 
data  and  analyses,  lessons,  and  recommendations,  summarized  in  this  report.  The  lessons 
address  considerations  of  acceptability,  impact,  supportability,  and  training  effectiveness 
assessment  itself.  Within  the  lessons,  the  research  proffers  solutions  to  the  identified  problems. 
Some  of  the  solutions  require  additional  research,  while  others  will  require  action  at  the  highest 
levels  of  Army  leadership.  Those  solutions  that  are  within  the  reach  of  training  designers  and 
developers  include  research  on  TSP  and  implementation  models,  redesign  of  TSP  products  and 
distribution  requirements,  and  planning  for  maintenance  and  sustainment  of  products.  Other 
solutions,  including  the  need  for  command  emphasis  at  division-level  and  higher,  and  the 
institutionalizing  of  the  products,  will  require  that  TRADOC  and  Army  leaders  make  a 
commitment  to  support  the  development  and  implementation  of  such  products  that  may  increase 
readiness  without  increasing  training  costs. 

Project  lessons  also  indicate  the  importance  of  maintaining  the  currency  of  training 
products,  and  of  providing  both  education  and  implementation  support  to  units  who  will  use  the 
products.  Additionally,  the  data  and  comments  from  users  highlight  the  importance  of  creating 
flexible  training  products  that  can  be  tailored  to  the  needs  of  the  user. 

Utilization  of  Findings: 

The  ISAT  project  has  generated  information  and  lessons  that  will  facilitate  the  fielding  of 
structured  training  products.  As  a  continuing  emphasis  is  placed  on  providing  low-resource, 
cost-effective  training  for  U.S.  Army  personnel,  this  report  can  lead  those  training  development 
efforts  into  the  selection  of  appropriate  design  and  implementation  initiatives. 
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Introduction 


Report 

organization 


The  U.S.  Army  is  facing  the  challenge  of  meeting  complex  training 
requirements  with  increasingly  constrained  resources.  As  a  result,  Army 
trainers  are  exploring  emerging  technologies  and  innovative  techniques 
that  can  be  used  to  achieve  training  goals.  Part  of  this  exploration  has  been 
accomplished  through  the  Force  XXI  Training  Program  (FXXITP). 

The  FXXITP  was  established  in  1995  by  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC)  and  is  overseen  by  the  Directorate  of 
Training  and  Doctrine  Development  (DTDD)  of  the  U.S.  Army  Armor 
Center  (USAARMC)  at  Fort  Knox,  Kentucky.  The  DTDD  has  worked 
with  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  to  research  and  develop  numerous  training  products  for  the 
program.  The  majority  of  the  work  was  performed  at  the  Armored  Forces 
Research  Unit  (AFRU)  at  Fort  Knox,  with  additional  work  completed  at 
the  Infantry  Forces  Research  Unit  (IFRU)  at  Fort  Benning,  Georgia.  The 
work  centered  on  the  development  of  simulation-based,  structured  training 
support  packages  (TSPs)  designed  for  the  collective  training  of  reserve 
component  (RC)  and  active  component  (AC)  battalion  and  brigade  staffs. 

Before  the  FXXITP  products  were  accepted  for  fielding,  the  TRADOC 
Deputy  Chief  of  Staff  for  Training  (DCST)  directed  that  they  be  assessed 
while  being  used  by  AC  brigades.  The  assessment  was  to  focus  on  the 
utility  of  the  TSPs  in  supporting  the  brigades’  yearly  training,  specifically 
in  preparation  for  National  Training  Center  (NTC)  rotations.  The  ARI  was 
to  provide  the  support  for  assessment  by  means  of  a  project  entitled 
Implementation  and  Support  for  the  Assessment  of  Force  XXI  Training 
Program  (ISAT).  With  a  team  of  contractor  personnel,  ARI  conducted  the 
assessment  with  the  assistance  of  four  AC  brigades  between  April  1998 
and  May  1999. 

While  the  primary  focus  of  the  DCST  directive  was  on  assessment  of 
FXXITP  product  utility,  ARI  directed  that  the  contractor  team  also  support 
implementation  of  the  products,  to  ensure  that  the  assessment  would  take 
place  in  a  context  of  full  product  use.  Therefore,  the  assessment  strategy 
included  plans  for  a  support  infrastructure  that  would  facilitate  integration 
of  the  FXXITP  products  into  unit  training  plans. 


This  research  report  describes  briefly  the  ISAT  project’s  assessment  of  the 
FXXITP  products.  The  bulk  of  the  report  is  devoted  to  summarizing  the 
implications  of  the  assessment  findings  and  recommendations.  A 
companion  report,  titled  Operational  Assessment  of  Force  XXI  Training 
Products:  Lessons  for  Successful  Fielding  (Pratt,  et  al.,  2000)  contains 
details  of  the  implementation  and  assessment,  as  well  as  more  complete 
presentation  of  data  analyses. 
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Background  Four  of  the  FXXITP  training  products  were  selected  to  be  the  focus  of  the 
ISAT  implementation  and  assessment: 

•  Battle  Staff  Training  System  (BSTS), 

•  Brigade  Staff  V ignettes 1 , 

•  Brigade  Staff  Exercises  (BSE),  and 

•  Brigade  and  Battalion  Staff  Exercises  (BBSEs). 

Research  and  development  on  the  BSTS  originated  at  ARI’s  Infantry 
Forces  Research  Unit  (Andre,  Wampler,  &  Olney,  1997).  Research  and 
development  that  led  to  production  of  the  brigade  staff  vignettes,  BSE,  and 
BBSE  was  conducted  by  ARI AFRU.  The  vignettes  and  the  BSE  were 
completed  in  1996  (C.  H.  Campbell,  Graves,  Deter,  &  Quinkert,  1998),  and 
the  BBSE  was  completed  in  1998  (C.  H.  Campbell  et  al.,  1999). 


Battle  Staff  The  BSTS  consists  of  functional  area  TSPs  for  individual  battalion-  and 

Training  System  brigade-level  staff  officers.  The  TSPs  combine  computer-based  instruction 
(CBI)  and  text.  Each  TSP  presents  a  course  of  instruction  by  means  of  CD- 
ROM  based  products  and  supplemental  text-based  instruction. 

Additionally,  a  training  management  system  (TMS),  Environment  for 
MultiMedia  interactive  instruction  (EMMii),  allows  the  trainer  or  other 
designated  individual  in  the  unit  to  monitor  the  progress  of  individual  staff 
officers  as  they  proceed  through  the  courses.  Courses  train  commanders 
and  staff  officers  in  their  individual  warfighting  skills  to  enhance  their 
proficiency  in  synchronization  of  battlefield  operating  systems. 


The  Brigade  Staff  The  brigade  staff  vignettes  are  24  short,  self-contained  training  activities 
Vignettes  that  focus  on  specific  staff  process  events.  Each  vignette  is  designed  to 

provide  practice  and  feedback  on  explicit  objectives  and  tasks.  The  TSP 
for  a  vignette  defines  the  objectives,  outcomes,  and  limits  of  the  training 
experience.  The  structure  also  includes  the  tactical  scenario  that  provides 
the  framework  for  the  required  activities.  Four  of  the  vignettes  use 
constructive  simulation  (Brigade/Battalion  Battle  Simulation  [BBS]  or 
Janus)  to  generate  scenario  events,  while  the  rest  use  only  live  simulation 
and  thus  can  be  conducted  in  a  classroom  or  office.  For  example,  the 
vignette,  “Conduct  Mission  Transition  -  Offense  to  Defense”  uses  Janus  to 
generate  and  feed  the  notional  battle  conditions  to  the  staff  in  their 
command  post  (CP).  The  vignette,  “Plan  Reconnaissance  and 
Surveillance,”  on  the  other  hand,  is  driven  by  predetermined  conditions  and 
has  no  need  for  a  dynamically  changing  situation. 


1  The  vignettes  were  later  renamed  Small  Group  Exercises  (SGEs)  by  DTDD. 
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The  Brigade  Staff 
Exercise 


The  Brigade  and 
Battalion  Staff 
Exercise 


The  BSE  components  include  integrated  scenarios  covering  the 
preparation,  planning,  execution,  and  consolidation  and  reorganization 
phases  of  the  battle.  The  BSE  scenarios  are  developed  for  constructive 
simulation  (BBS)  and  are  conducted  in  the  notional  Mojave  Desert, 
including  the  NTC  terrain  “box.”  The  TSP  for  the  exercise  also  contains 
brigade  staff  preparation  materials  and  guidance,  simulation  system 
electronic  files  and  documentation,  instructions  for  interactors  and 
roleplayers,  materials  and  guidance  to  support  observation  and  feedback, 
and  complete  details  for  exercise  conduct  and  control. 

The  BSE  focuses  on  the  performance  of  the  brigade  commander  and 
selected  staff  members.  There  are  several  key  emphases,  including  the 
process  of  staff  planning  and  decision-making;  the  integration  of  all 
brigade  assets  within  the  combat,  combat  support  (CS),  and  combat  service 
support  (CSS)  arenas;  and  the  interactions  among  the  primary  staff 
members  during  planning,  preparation,  execution,  and  consolidation  and 
reorganization. 


The  BBSE  is  a  structured,  multiechelon,  battlestaff  training  exercise  that 
encompasses  planning,  preparation,  execution,  and  sustainment.  It  allows 
battalion  and  brigade  commanders  and  staffs  to  interact  as  they  plan  for 
tactical  missions,  employ  combat  power,  and  conduct  rear  area  sustainment 
operations.  Like  the  BSE,  it  takes  place  in  a  BBS-simulated  Mojave  desert 
setting.  A  training  scenario  generates  an  NTC-like  operational  tempo 
requiring  24-hour  operations  and  stresses  concurrent  and  parallel  planning 
processes.  The  exercise  allows  the  unit  to  practice  and  receive  feedback  on 
key  staff  procedures,  which  enable  the  unit  to  enter  major  field  training 
exercises  (FTX)  at  a  higher  level  of  proficiency  in  the  staff  process.  The 
BBSE  generates  the  information,  cues,  and  simulated  operations  that  allow 
CSS  to  be  a  major  consideration. 

The  BBSE  differs  from  the  BSE  in  several  aspects.  It  presents  a  faster 
battle  tempo,  including  a  more  robust  opposing  force  (OPFOR),  concurrent 
processes,  24-hour  operations,  and  deployed  CPs  for  the  brigade  and 
battalion  staffs;  it  incorporates  multiechelon  staff  activities  and  staff 
section  activities;  and  the  performance  objectives  are  much  broader, 
usually  encompassing  techniques  and  procedures  for  multiple  battlefield 
functions. 
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Project 

Objectives 


The  vision  of  the  project’s  procedures  and  outcomes  was  presented  by  ARI 
in  terms  of  two  major  objectives: 

•  To  build  an  implementation  support  infrastructure  at  participating 
unit  locations  to  facilitate  the  assessment. 

•  To  conduct  the  implementation  and  assessment  at  unit  locations 
and  Fort  Irwin,  California  (the  NTC). 


The  project’s  Execution  Plan  (HumRRO  et  al.,  1998b)  outlined  the 
project’s  strategy  for  accomplishing  the  objectives.  It  included  plans  for 
managing  the  project  from  Fort  Knox,  and  providing  support  to  the  user 
units  at  other  locations.  Data  collection  for  the  assessment  would  involve 
interviews,  questionnaires,  and  observations,  as  outlined  in  the  Assessment 
Plan  (HumRRO  et  al.,  1998a). 


Limitations  on  the  Because  only  two  brigades  were  going  to  participate  in  the  implementation 
assessment  and  assessment,  and  because  there  was  no  opportunity  for  rigorous  control 

of  the  brigade  activities,  the  assessment  could  not  serve  as  a  full  evaluation 
of  the  products’  value,  effectiveness,  or  impact.  Instead,  ARI  and  DTDD 
asked  for  program  monitoring,  in  the  manner  of  a  complete  case  study  of 
the  implementation  activities,  with  intense  efforts  to  obtain  reactions  and 
suggestions  from  all  those  involved  in  use  of  the  products. 


The  ISAT  Project 
execution  and 
assessment 
approach 


Implementation 
Support  Plan 


As  a  general  approach,  the  ISAT  project’s  assessment  focused  on  the 
potential  success  of  the  FXXITP  products  in  the  hands  of  the  users.  The 
basic  strategy  was  to  monitor  users  while  they  worked  with  the  training 
products,  and  to  gather  feedback  regarding  suitability,  acceptability, 
training  effectiveness,  outcomes,  and  supportability.  Focusing  on  the 
impact  of  the  training  products  rather  than  on  their  quality  distinguished 
the  ISAT  assessment  from  the  established  formative  evaluation  approach 
used  in  previous  training  development  research  (e.g.,  C.  H.  Campbell, 
Deter,  &  Quinkert,  1997). 


Assessing  impact,  however,  demands  either  an  assumption  or  an  assurance 
of  product  use.  The  products  were  still  unfielded,  and  there  was  no 
history  of  product  use  by  units,  other  than  the  formative  evaluation  trials 
without  further  mentoring  or  assistance.  Therefore,  ARI  directed  that  the 
ISAT  team  assist  as  necessary  to  ensure  that  the  products  could  be  used, 
and  monitor  all  aspects  of  the  attempted  implementation. 
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Anticipated 

implementation 

disruptions 


Unit  participation 


A  known  complication  was  that  the  participating  brigades  would  be 
preparing  for  a  rotation  at  the  NTC.  While  the  impending  NTC  rotation 
greatly  increased  the  potential  for  direct  feedback  on  the  effectiveness  of 
the  training  as  implemented,  the  schedule  for  such  a  train-up  is  intense. 

As  a  result,  the  FXXITP  products  would  likely  compete  with  events  from 
existing  unit  strategies  routinely  used  to  prepare  for  NTC  rotations. 

Within  the  ISAT  project,  the  intent  was  to  work  with  the  units  and  provide 
recommendations  that  would  allow  the  products  to  complement  the  units’ 
training  strategies.  This  would  leverage  use  of  the  training  products  to 
better  prepare  the  units  to  adjust  to  the  faster-paced  training  environment 
associated  with  an  NTC  rotation.  The  ISAT  approach  emphasized  a  high 
level  of  support  during  preparation  and  conduct  of  training  exercises  to 
increase  the  likelihood  that  units  would  be  able  to  use,  and  would  use,  the 
products  being  assessed. 

Despite  the  support  from  the  ISAT  team,  it  was  anticipated  that  use  of  the 
products  could  be  affected  by  outside  factors  such  as  operational 
contingency  missions  (e.g.,  Intrinsic  Action,  Team  Spirit),  weather-related 
disasters,  and  shortfalls  of  training  funds.  These  conditions  could  cause 
the  units  to  alter  the  way  they  used  the  products.  The  accommodations 
required  by  such  disruptions  would  be  instructive  from  a  case  study  and 
lessons  learned  perspective,  but  it  was  acknowledged  that  they  could  also 
affect  the  validity  of  the  program  assessment. 


As  the  ISAT  project  began,  TRADOC  DCST  coordinated  with  the  U.S. 
Forces  Command  (FORSCOM)  to  confirm  the  tasking  of  units  to  support 
the  assessment.  The  TRADOC  DCST  had  directed  that  the  assessment  be 
conducted  with  two  brigades  within  III  Corps.  The  brigades  were  to  be 
preparing  for  their  NTC  rotations  as  this  would  best  accommodate  the 
long-term  ISAT  assessment.  One  of  the  two  brigades  initially  selected 
asked  to  be  replaced,  citing  their  then  advanced  preparation  status  for  their 
NTC  rotation.  Another  brigade  then  agreed  to  participate  upon  the 
condition  that  they  would  assist  in  the  assessment  of  only  those  products 
that  would  meet  their  peculiar  training  needs.  These  products  included  the 
BSTS  and  vignettes,  but  not  the  BSE  and  BBSE. 

By  May  1998,  one  of  the  brigades  had  begun  to  use  the  FXXITP 
materials.  At  that  time,  however,  they  were  ordered  to  prepare  for 
deployment.  Once  this  occurred,  the  brigade  could  no  longer  support  the 
ISAT  effort.  In  July  1998,  yet  another  brigade  volunteered  to  participate 
in  the  assessment.  The  participating  units,  then,  included  two  brigades, 
one  of  which  would  not  be  using  the  BSE  or  the  BBSE.  A  limited  amount 
of  use  had  also  occurred  in  the  first,  deployed,  brigade. 
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Scope  of  support 


Implementation 
support  teams 


Planning  and 
preparation 


Reports  on  the  development  of  the  FXXITP  products  have  indicated  that 
first-time,  and  even  subsequent,  implementations  of  the  products  have 
required  a  fair  amount  of  implementation  support  from  personnel  familiar 
with  the  products  (Andre  et  al.,  1997;  C.  H.  Campbell  et  al.,  1998;  C.  H. 
Campbell  et  al.,  1999;  Graves,  Campbell,  Deter,  &  Quinkert,  1997; 
Hoffman,  Graves,  Koger,  Flynn,  &  Sever,  1995).  Although  one  of  this 
project’s  critical  assessment  issues  involved  investigating  the  extent  to 
which  units  could  use  the  products  without  external  support,  it  was 
evident  that  such  support  would  be  required  to  facilitate  the  assessment. 
As  a  result,  ARI  directed  that  support  be  offered  and  available  as  needed, 
especially  during  unit  planning  and  preparation  for  product  use. 


The  IS  AT  team  established  to  support  product  use  included  training 
developers  and  military  subject  matter  experts  (SMEs)  who  had  developed 
the  FXXITP.  The  project  located  a  Training  Support  Coordinator  (TSC) 
on-site  for  the  units  participating  in  the  assessment.  The  TSC  coordinated 
and  directed  project  activities  at  the  site.  He  mentored  and  assisted  the 
units  while  he  oversaw  and  coordinated  the  training  support  for  the 
exercises.  He  also  assisted  with  the  initial  orientation  training  to  the  corps 
and  division  leaders,  provided  the  training  for  brigade  leaders,  and 
directed  the  training  for  brigade  and  battalion  training  managers.  He  was 
directly  responsible  for  carrying  out  requirements  of  the  assessment  plan 
at  that  location. 


During  the  planning  and  preparation  phase,  there  was  an  intense  proactive 
involvement  with  the  participating  brigades.  That  involvement  began 
with  a  thorough  informational  program  for  leaders,  training  managers, 
training  support  personnel  and  users  of  the  various  products.  Also  during 
the  initial  training  periods,  the  IS  AT  staff  briefed  the  simulation  center 
personnel  on  the  FXXITP  products.  The  simulation  center  personnel  were 
trained  to  implement  BSTS  and  monitor  vignettes  if  other  units  at  that 
location  wanted  to  use  them. 
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Train  the  trainer 

sessions 


TSC  assistance 


Surge  team 
assistance 


Rapid  response 

training 

development 


The  orientation  sessions  were  followed  by  train  the  trainer  (T3)  sessions, 
provided  to  the  participating  brigades  prior  to  their  initial  use  of  the 
FXXITP  products.  The  T3  in  preparation  for  BSTS  included  both  user 
orientation  and  training  for  unit  BSTS  administrators.  Vignette  T3 
sessions  also  had  two  parts,  addressing  the  unit  training  coordinators  and 
the  unit  staff  members  who  would  participate  as  the  training  audience. 

The  T3  for  the  BSE  and  BBSE  was  modeled  after  the  T3  program 
incorporated  in  the  BBSE  TSP  itself.  It  was  preceded  by  informational 
sessions  for  the  commanders  and  brigade  staffs  and  the  battle  simulation 
staffs.  The  three-day  T3  session  itself  was  decentralized  and  focused  on 
the  separate  groups  who  would  support  the  exercises,  including  simulation 
controllers,  the  division  response  cell,  the  OPFOR  controllers,  and  the 
training  observers. 


Subsequent  to  the  initial  training,  the  ISAT  team  TSCs  continued  close 
coordination  with  and  support  of  the  participating  brigades  and  battalions 
as  they  planned  for  and  used  the  products.  They  provided  advice  and 
assistance  to  the  staffs  of  the  units  in  selecting  and  implementing  the 
appropriate  option  from  those  available  within  the  products. 


When  one  of  the  participating  brigades  prepared  to  conduct  a  BSE  or 
BBSE,  a  surge  team  from  the  ISAT  staff  would  travel  to  the  brigade 
location  to  assist  with  the  event.  At  that  time,  they  would  monitor  and 
assist  with  training  for  a  wide  range  of  participants  (e.g.,  observers, 
roleplayers  portraying  enemy  and  friendly  units,  and  simulation 
controllers)  and  assist  with  the  conduct  of  the  exercise. 


One  additional  duty  of  the  ISAT  team,  not  anticipated  in  the  execution 
plan,  involved  rapid-response  training  development.  The  need  arose  when 
one  unit  asked  the  ISAT  team  to  provide  a  modified  exercise  that  would 
provide  benefits  similar  to  those  of  the  BSE  but  would  be  implemented 
using  Janus  rather  than  BBS. 
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Product  As  a  general  rule,  the  units  implemented  the  FXXITP  training  products  in 

Utilization  progressive  fashion.  Training  began  with  the  BSTS  products,  then 

progressed  to  the  vignettes,  and  (for  one  brigade)  culminated  with  the 
BBSE  just  prior  to  their  NTC  rotation.  The  implementation  events  and 
variances  from  the  intended  plan  are  described  in  Pratt  et  al.  (2000). 


Utilization  of  the  The  ISAT  team  supported  and  assessed  three  implementations  of  the 
Battle  Staff  Training  BSTS.  Each  implementation  was  different  with  respect  to  hardware 
System  configuration,  T3  comprehensiveness,  BSTS  system  training,  and  unit 

member  usage.  In  general,  usage  was  sporadic,  and  not  widely  mandated 
nor  encouraged  by  the  chain  of  command. 


Utilization  of  the  One  brigade  conducted  only  one  vignette,  with  the  assistance  of  the  T3 
Vignettes  trainer  and  in  conjunction  with  their  T3  session.  The  second  brigade 

conducted  three  vignettes,  but  made  significant  modifications  to  the 
scenario.  The  third  brigade  conducted  two  vignettes,  but  did  not  follow 
all  of  the  TSP-specified  procedures  for  unit  preparation. 


Utilization  of  the 
Brigade  Staff 
Exercise 


None  of  the  brigades  implemented  the  BSE.  One  brigade  worked  with  the 
ISAT  team  to  produce  a  Janus-based  counterpart  exercise,  but  a  complete 
TSP  was  not  developed  for  the  exercise.  Instead,  experienced  ISAT  team 
members  filled  in  specific  roles  to  support  the  implementation. 


Utilization  of  the 
Brigade  and 
Battalion  Staff 
Exercise 


One  brigade  conducted  a  BBSE,  mid-way  between  their  leader  training 
program  (LTP)  visit  and  NTC  rotation.  Several  elements  external  to  the 
brigade  supported  the  preparation  and  execution  of  the  BBSE,  including 
the  division  headquarters  and  a  sister  brigade.  The  installation’s 
simulation  center  provided  BBS  support  personnel  as  well  as  preparation 
support  for  the  exercise.  Members  of  the  ISAT  team  provided  substantive 
support  during  train-up,  but  served  only  as  data  collectors  during  the 
BBSE  itself. 


LTP  participation  The  ISAT  team  was  able  to  follow  only  one  brigade  to  the  LTP  at  Fort 

and  NTC  rotation  Irwin,  and  to  the  subsequent  NTC  rotation;  this  was  the  same  brigade  that 

had  conducted  three  vignettes,  the  Janus  exercise,  and  the  BBSE. 


Summary  As  should  be  obvious,  the  implementation  did  not  proceed  according  to 

plan.  The  differences  between  the  plan  and  the  reality  are  important  in 
interpreting  the  findings.  No  blame  can  be  attached  to  any  units,  but  the 
fact  remains  that  the  implementation  assessment  was  compromised  by  the 
shortfall. 
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Assessment 

Plan 


The  IS  AT  project  represented  an  important  milestone  for  ARI’s  training 
development  efforts.  As  the  first  assessment  of  the  FXXITP  training 
products  in  a  routine  operating  environment,  the  effort  would  provide  an 
opportunity  to  gather  valuable  data  regarding  three  main  areas  of  interest: 


•  Acceptability — Were  the  training  materials  doctrinally  correct? 
Were  they  usable?  Were  other  materials  necessary? 

•  Perceptions  of  Impact — Were  the  training  products  useful  for 
learning  and  practicing  job  requirements  and  preparing  for  other 
major  training  events?  Did  users  think  that  the  training  had  (or 
would  have)  a  positive  effect  on  performance? 

•  Supportability — What  would  it  take  to  make  the  training  products 
useable  within  a  brigade’s  training  plan? 


Data  collection 
methods 


A  principal  goal  of  the  assessment  was  to  collect  data  at  every  opportunity 
at  which  the  participating  units  used  the  various  training  products. 
Throughout  the  course  of  the  ISAT  project,  there  were  six  major  events 
that  were  expected  to  provide  opportunities  for  data  collection: 

•  BSTS  use 

•  Vignette  use 

•  Conduct  of  the  BSE/Janus  Exercise 
.  NTC  LTP 

•  Conduct  of  the  BBSE 

•  NTC  Rotation 

ISAT  team  members  were  to  serve  as  on-site  data  collectors  in  both  the 
home  station  and  NTC  arenas. 

An  additional  occasion  for  obtaining  observations  about  the  FXXITP 
products  came  when  the  ISAT  team  was  invited  to  brief  the  four  products 
(BSTS,  vignettes,  BSE,  and  BBSE)  for  personnel  from  the  Seventh  Army 
Training  Center  (7th  ATC).  The  briefing  included  information  sessions  on 
each  product,  walkthroughs  of  the  TSPs,  and  demonstrations  of  the 
training  exercises.  The  7th  ATC  representatives  provided  impressions  and 
feedback  based  on  the  information  and  demonstrations. 


The  ISAT  assessment  was  designed  to  incorporate  a  wide  range  of  data 
collection  techniques,  including  user  ratings,  written  and  oral  comments, 
observations,  and  descriptive  documentation.  By  using  multiple  methods 
to  address  each  information  need,  corroborating  or  clarifying  data  would 
be  captured. 
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Assessment 

Events 


Data  reduction  and 
analysis  procedures 


Summary 


Lessons 

Learned 


As  described  earlier,  there  were  many  departures  from  the  implementation 
plans  within  both  of  the  participating  brigades.  As  a  result,  the  data 
collection  was  less  comprehensive  than  had  been  anticipated,  and  results 
were  not  easily  interpretable.  However,  these  difficulties  had  been 
anticipated  and  discussed  among  DTDD,  ARI,  and  ISAT  team 
researchers,  and  various  corrective  actions  were  taken  in  order  to  glean  the 
greatest  amount  of  usable  information  for  the  three  areas  of  interest. 


In  preparation  for  the  data  reduction  and  analysis,  a  database  was 
constructed  that  accommodated  the  majority  of  the  collected  data.  As  the 
data  were  collected  from  the  training  sites,  the  ISAT  team  entered  them 
into  the  computer  database. 

The  ISAT  team  analyzed  the  assessment  data  in  two  stages:  quick-look 
processing  of  data  as  they  became  available,  and  comprehensive  analysis 
and  interpretation  of  cumulative  data.  The  goal  of  this  strategy  was  to 
identify  key  findings  in  a  timely  manner  while  reserving  systematic 
conclusions  for  thorough  analysis. 


This  section  has  described  the  purposes,  objectives,  and  methods  of  the 
ISAT  project’s  assessment  of  the  FXXITP  products.  The  assessment, 
focusing  on  product  impact,  was  intended  to  be  conducted  in  a  routine 
operating  environment  and,  thus,  relied  on  the  integration  of  the  FXXITP 
products  into  unit  training  plans.  The  effects  of  competing  demands  on 
unit  time,  resources,  and  focus  on  product  implementation  were  a 
significant  confounding  issue. 


Observation  of  the  attempted  implementation  revealed  a  wide  range  of 
issues  concerning  the  usability  of  the  FXXITP  products.  This  section 
discusses  those  issues  from  the  perspective  of  lessons  learned  during  the 
project  and  recommendations  that  emerged  in  discussions  with  members 
of  the  participating  units.  The  lessons  are  grouped  in  three  areas: 
acceptability,  perceptions  of  impact,  and  supportability.  A  final  lesson 
addresses  training  effectiveness  assessment  itself. 
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Acceptability  During  this  assessment  acceptability  was  defined  as  .  .the  unit’s 

acceptance  of  the  FXXITP  products  as  being  doctrinally  correct  and 
containing  usable  materials.”  Questions  that  were  addressed  included: 
Were  the  training  materials  doctrinally  correct?  Were  they  usable?  Were 
other  materials  necessary? 

The  acceptability  of  the  FXXITP  products  was,  for  the  most  part,  positive. 
BSTS  users  indicated  that  the  BSTS  met  the  basic  needs  for  individual 
staff  level  training,  although  it  was  not  sufficiently  interactive.  The 
vignettes  were  perceived  as  sufficiently  flexible  for  use  in  learning 
centers,  in  the  field,  or  for  “Thursday  morning”  (staff  development) 
training.  The  BSTS  and  BSE  were  found  by  7th  ATC  reviewers  to  be  the 
least  current  of  the  products.  Most  felt  that  the  products  supported  the 
training  they  were  intended  to  support.  Yet  many  comments  indicated 
that  there  were  changes  that  needed  to  be  made  to  make  the  products  more 
useful  and  keep  them  viable. 

Six  lessons  addressing  acceptability  were  formulated,  as  described  below. 


Lesson  Al:  Structured  training  products  will  inevitably  become  outdated,  and  will  require 
continual  examination  and  updating. 

It  took  over  five  years  to  develop,  implement,  and  assess  the  FXXITP 
products.  Because  Army  doctrine  has  continued  to  evolve,  some  of  the 
products  are  currently  outdated,  doctrinally  and  technologically.  Most  of 
the  shortcomings  of  the  FXXITP  products  deal  with  doctrinal  changes  to 
terms  and  symbols,  with  some  changes  to  tactics,  techniques,  and 
procedures,  and  newer  ways  to  deliver  information.  This  is  an  inevitable 
occurrence  for  any  structured  training  product,  and  underlies  one  of  the 
reasons  that  the  program  must  have  Army-wide  support. 


Lesson  A2:  Even  though  it  is  inevitable  that  the  products  will  become  outdated,  they  can  be  used 
while  updating  goes  on. 

Despite  the  doctrinal  shortcomings  of  the  products,  the  units  were  still 
able  to  use  the  products  for  training.  Reviewers  of  the  BSTS  from  the 
project,  user  units,  and  7th  ATC  identified  many  instances  of  outdated 
doctrine;  yet  users  of  BSTS  were  consistently  positive  regarding  the 
doctrinal  currency  of  the  BSTS  courses.  After  their  LTP,  the  one  brigade 
still  participating  indicated  that  the  training  had  been  useful. 

Continued  on  next  page 
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Lesson  A2  ( Continued):  Even  though  it  is  inevitable  that  the  products  will  become  outdated,  they 
can  be  used  while  updating  goes  on. 

When  asked  about  the  apparent  discrepancy,  one  reviewer  said  that  the 
fighting  units  were  more  likely  to  overlook  doctrinal  inconsistencies  if  the 
potential  training  benefit  was  high.  In  other  words,  they  cared  more  about 
learning  to  fight  well  than  how  they  learned  how  to  fight  well.  When 
users  commented  on  the  BSTS,  they  were  more  vocal  about  the  potential 
benefit  the  training  could  offer  than  the  slight  discrepancies  in  doctrine. 
While  doctrinal  consistency  is  critical,  it  may  be  that  intermittent  updates, 
and  not  a  continuously  ongoing  revision  process,  are  good  enough.  The 
fact  is  that  (as  stated  in  the  previous  lesson)  all  structured  training 
products  that  are  tied  to  doctrine  will  become  outdated  as  doctrine 
changes.  But  the  Army,  or  TRADOC,  or  the  FXXITP,  must  find  a  way  to 
distinguish  between  doctrinal  inconsistencies  that  lead  to  bad  training,  and 
those  that  are  below  the  noise  level.  It  seems  apparent  that  training  units 
are  willing  and  able  to  overlook  flaws  of  products  if  those  products  are 
perceived  to  be  good  training  products. 

It  seems  that,  in  the  short  run,  the  FXXITP  products  can  meet  the  training 
needs  despite  their  doctrinal  shortcomings.  However,  units  will  quickly 
become  tired  of  training  with  outdated  products  and  will  either  quit  using 
them  or  be  required  to  update  the  products  themselves.  In  either  case,  the 
Army  will  have  lost  all  of  the  advantages  of  standardized  and  structured 
training.  To  ensure  that  these  products  remain  viable  to  the  unit  while 
remaining  standardized  and  structured,  program  resources  must  be 
allocated  to  gather  feedback  on  these  products,  review  them  for  currency, 
and  update  them  in  a  timely  manner.  The  force  must  be  kept  informed  of 
the  updating  intentions  and  progress. 

One  way  to  facilitate  the  maintenance  of  doctrinal  currency  would  be  to 
use  the  same  products  for  institutional  training.  By  doing  so,  the  training 
institutions  (which  are  also  responsible  for  updating  doctrine)  will  by  the 
nature  of  their  institutional  training  requirements  focus  on  product 
currency.  This  also  provides  an  advantage  of  more  widespread 
understanding  of  the  products,  since  unit  leaders  will  have  used  them 
during  their  institutional  training. 
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Lesson  A3:  Future  TSPs  can  be  more  usable  if  we  incorporate  more  of  the  human  dimensions  of 
TSP  usage. 

A  consistent  finding  was  that  even  with  TSCs,  surge  team  assistance,  and 
T3  sessions,  many  participants  indicated  that  they  did  not  use  their 
materials.  Unit  trainers  commented  that  it  was  difficult  to  reproduce, 
distribute,  and  explain  the  contents  of  the  materials.  The  BSTS  data  (or 
rather,  the  lack  of  data)  indicated  clearly  the  prevalence  of  problems  with 
training  management  components  that  often  left  unit  leaders  only  partially 
aware  of  product  options  and  value,  and  lacked  safeguards  to  prevent  loss 
of  data. 

Such  findings  indicate  the  products  did  not  fully  meet  the  working  needs 
of  the  training  audience.  One  likely  cause  is  that  developers  have  not  yet 
identified  effective  ways  to  address  training  constraints  with  the  same 
expertise  that  is  applied  to  task-based  training  needs.  Adjusting  the  design 
process  to  bring  the  human  dimensions  of  TSP  usage  into  closer  balance 
with  training  objectives  and  technology  considerations  could  lead  to 
products  that  make  the  information  easier  for  the  users  to  access  and 
reduce  the  burden  of  getting  ready  to  train. 

This  may  indicate  a  need  to  reexamine  the  entire  implementation  model. 

It  is  easy  to  say  that  we  must  streamline  the  presentation  of  materials  and 
minimize  the  load  imposed  on  the  trainers  and  training  audience.  At  the 
same  time,  it  is  imperative  that  crucial  information  (e.g.,  training 
objectives,  definition  of  training  audience  versus  support  personnel, 
specific  assignments)  be  disseminated  in  advance,  and  that  all  necessary 
information  and  guidance  that  defines  the  product  structure  be  available. 
Failure  to  provide  clear  information  in  advance  inevitably  leads  to 
confusion  about  the  exercise’s  purpose,  roles  and  duties,  which  in  turn 
degrades  the  event’s  training  value. 

Developers  of  structured  training  products  have  expressed  frustration  with 
the  lack  of  complete  use  of  TSP  materials  for  years  (C.  H.  Campbell  et  al., 
1998;  C.  H.  Campbell  et  al.,  1999;  Graves  et  al.,  1997).  Each  new 
development  attempts  to  provide  TSPs  that  are  more  user-friendly,  yet  we 
have  not  yet  cracked  the  code  on  user  needs.  We  have  simply  not  been 
able  to  comprehend  the  obstacles  and  incorporate  effective  solutions.  The 
answer  will  not  be  a  simple  one.  It  is  likely  to  be  a  combination  of 
additional  research  on  what  works  for  time-stressed  units.  Army  and 
command  emphasis  on  use  of  the  products,  and  the  availability  of  surge 
teams  or  support  coordinators.  As  yet,  we  have  not  been  able  to  conduct  a 
controlled  study  of  the  effectiveness  of  different  TSP  approaches.  Such 
considerations  may  provide  a  substantial  payoff  to  both  training 
developers  and  researchers. 
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Lesson  A4:  There  needs  to  be  a  balance  between  the  time  allotted  to  prepare  for  these  exercises 
and  the  time  it  takes  to  prepare. 

These  FXXITP  products  are  not  advertised  as  “turn-key”  training,  which 
would  convey  a  design  characteristic  that  is  supposed  to  enable  the 
training  coordinators  to  implement  the  training  products  with  minimal 
preparation.  Nonetheless,  unit  training  coordinators  rarely  allocate 
sufficient  time  for  preparation.  Time  has  proven  to  be  the  most  valuable 
commodity  to  the  units  participating  in  this  assessment,  and  is  probably 
equally  valuable  across  AC  and  RC  units.  As  a  general  rule,  personnel  did 
not  study  the  guides  or  accomplish  preparation  activities  in  accordance 
with  the  TSP  instructions.  It  is  easy  for  us  to  blame  the  TSPs  when  in  fact 
it  may  be  that  the  unit  is  not  providing  enough  time  to  prepare  for  training. 


Lesson  A5:  In  addition  to  doctrinal  updates,  TRADOC  and  the  FXXITP  should  continue  to 
incorporate  technology  updates. 

This  lesson  emerged  as  the  team  observed  BSTS  and  BBSE 
implementation.  Units  in  this  assessment  used  BSTS  courseware  in  stand¬ 
alone  mode.  However,  the  courses  were  designed  primarily  for  a  local 
area  network  (LAN)  operating  environment.  This  is  not  the  LAN 
environment  we  are  used  to  using  today,  but  a  hard-wired  Ethernet  LAN 
that  was  commonplace  in  1993.  Similarly,  the  BBSE  was  originally 
written  for  a  version  of  BBS  software  that  was  no  longer  being  used  at  the 
time  of  implementation.  The  ISAT  team  assisted  the  simulation  center 
personnel  with  the  updating  to  allow  the  exercise  to  occur.  This  shows  a 
need  to  validate  assumptions  about  infrastructure  in  place  when  products 
are  ready  for  fielding.  Although  a  comprehensive  systems  approach 
builds  infrastructure  in  parallel  with  course  development,  the  technology 
needs  to  be  reexamined  continually,  just  as  do  doctrinal  foundations. 


Lesson  A6:  TRADOC  and  the  FXXITP  should  continue  to  develop  ways  for  units  to  tailor  TSP 
materials  to  their  unit’s  organization  and  equipment,  training  audience,  and  Standing  Operating 
Procedure  (SOP). 


This  lesson  addresses  a  range  of  issues  on  the  advantages  of  structure  and 
standardization,  balanced  against  the  need  for  programs  that  are  flexible 
enough  for  valuable  training.  Except  for  the  BSTS,  TSPs  could  be 
provided  in  an  electronic  form  so  that  the  unit  can  modify  the  unit 
information  to  match  their  own.  Another  option  would  be  to  deliver  these 
TSPs  using  the  model  of  the  Commanders’  Integrated  Training  Tool 
(CITT  [Gossman  et  al.,  1999])  This  computer-  and  Internet-based  system, 
currently  under  development  by  ARI  and  Simulation  Training  and 
Instrumentation  Command,  is  designed  to  help  units  use  existing  TSPs  as 
written,  modify  them  for  their  own  situations,  or  develop  new  TSPs. 

Continued  on  next  page 
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Lesson  A6  (continued):  TRADOC  and  the  FXXITP  should  continue  to  develop  ways  for  units  to 
tailor  TSP  materials  to  their  unit’s  organization  and  equipment,  training  audience,  and  SOP. 

The  current  approach  of  designing  complete  structured  TSPs  to  be  used  by 
a  diverse  mix  of  tactical  units  leads  unit  personnel  to  adopt  alter-identities, 
notional  equipment  mixes,  and  so  on.  The  desire  to  replicate  exactly  the 
unit’s  organization,  equipment,  signal  operations  instructions,  tactical 
SOP,  and  other  operational  conditions  is  understandable,  but  there  are  no 
data  showing  that  precise  replication  increases  training  value  or  transfer  of 
training.  Research  to  evaluate  this  dimension,  as  well  as  to  find  ways  to 
make  the  products  more  flexible,  would  help  determine  if  the  Army 
should  invest  in  unit-specific  TSPs. 

Units  participating  in  this  assessment  consistently  modified  the  specified 
procedures.  Abbreviated  preparation  activities,  training  audience  changes, 
altered  missions,  scheduling  peculiarities,  self-generated  performance 
assessment  and  after  action  review  (AAR)  procedures,  and  numerous 
other  alterations  were  common.  This  highlights  the  importance  of 
designing  TSPs  to  provide  substantial  execution  flexibility.  For  example, 
the  vignette  TSPs  could  include  instructions  for  integrating  extra 
participants  and  outfitting  them  with  relevant  training  materials.2  The 
design  process  should  take  into  account  the  likely  variations  in  utilization, 
and  the  evaluation  process  should  incorporate  program  flexibility  as  a 
major  dimension  of  interest. 

The  single  implementation  of  the  BBSE  is  a  case  in  point.  The  product 
was  altered  significantly  to  meet  the  unit’s  perceived  training  needs  in 
preparation  for  an  NTC  rotation.  Yet  the  BBSE  exercise  director,  the 
installation  deputy  commanding  general,  pointed  out  that  “If  we  had  to 
generate  this  exercise  on  our  own,  given  our  organization  and  other 
requirements,  we  would  not  have  done  as  well.” 


2  The  reader  should  notice  at  this  point  the  insidious  subtlety  of  “good  idea  creep.”  Every  such  good  idea 
increases  the  bulk  of  the  TSP,  and  we  have  yet  to  discover  how  to  include  the  many  good  ideas  without 
complicating  TSP  delivery  even  further. 
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Perceptions  of  Impact,  in  this  research,  included  issues  of  learning,  job  performance,  and 

impact  organizational  impact.  Questions  of  interest  included:  Were  the  training 

products  useful  for  learning  and  practicing  job  requirements  and  preparing 
for  other  major  training  events?  Did  users  think  that  the  training  had  (or 
would  have)  a  positive  effect  on  performance?  Most  of  the  responses 
from  users  indicated  that  they  saw  the  potential  training  value  of  the 
products.  After  the  LTP  experience,  brigade  members  said  that  the 
products  had  helped  them  prepare;  after  the  NTC  rotation,  their  comments 
showed  that  they  were  beginning  to  see  some  value  in  the  products.  Many 
of  the  answers  to  these  questions  focused  less  on  the  existing  products 
than  they  did  on  what  was  still  needed — a  subtle  tribute  to  the  perceived 
training  impact.  Only  one  lesson  was  derived. 


Lesson  PI1:  The  existing  products  need  to  be  expanded. 

The  call  for  additional  expanded  products  came  mostly  on  the  BSTS  and 
vignettes.  On  the  BSTS,  users  requested  courses  for  their  staff  section 
members  as  well  as  for  the  staff  officers.  While  there  are  cases  where 
staff  officers  have  used  the  BSTS  courses  for  their  sections,  specific 
courses  would  provide  the  information  and  training  that  noncommissioned 
officers  (NCOs)  need. 

On  vignettes,  users  commented  that  brigade  staff  vignettes  are  not  enough. 
The  units  also  need  vignettes  for  battalion  staffs,  for  brigade  leadership, 
for  leaders  linked  from  brigade  through  company  and  for  commanders  and 
their  staffs.  The  so-called  “Leader  Vignettes”  are  needed  to  ensure  that  all 
staff  section  leaders  and  commanders  within  the  brigade  combat  team  can 
develop  a  common  understanding  of  the  brigade  commander’s  intent  and 
information  needs.  The  multi-echelon  leader  vignettes  would  help  the 
subordinate  commanders  understand  the  amount  and  type  of  detail  the 
brigade  commander  needs  to  make  a  decision.  These  would  likely  be 
associated  closely  with  the  commander’s  critical  information  requirements 
and  Decision  Support  Template. 

The  goal  of  these  vignettes  would  be  to  provide  a  concrete  experience 
(adult  learning  model  start  point)  from  which  the  participants  could 
discuss  the  “art”  of  warfighting.  Similarly,  the  commander-staff  vignettes 
would  bring  the  key  leaders  of  the  staff  into  the  process  where  the  staffs 
add  clarity  to  the  information  provided  by  the  commanders  because  they 
have  the  time  to  analyze  the  information.  One  user  explained  that  the 
outcomes  of  the  vignettes  for  leaders  should  be  staff  recommendations  for 
commanders;  the  vignette  should  train  staff  leaders  to  analyze  the  right 
amount  of  information  quickly. 
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Supportabiiity  Supportability  of  the  products  referred  to  whether  a  brigade  would  be  able 

to  conduct  the  FXXITP  exercises  within  the  resources  available  to  them. 
The  key  question  was:  What  would  it  take  to  make  the  training  products 
useable  within  a  brigade’s  training  plan?  In  this  regard  each  component 
of  the  FXXITP  had  to  be  evaluated  for  supportability  based  on  the 
requirements  established  for  its  use,  as  laid  out  in  the  TSP. 

The  experience  on  this  project  would  indicate  that  all  of  the  assessed 
components  of  the  FXXITP  require  external  support  for  unit  use.  The 
amount  of  support  will  vary.  For  the  BSTS,  both  system  installation  and 
initial-use  training  were  provided,  and  were  later  supplemented  with 
continuing  assistance  in  recording  usage  data  and  keeping  the  systems 
running.  The  support  for  the  vignettes  was  relatively  slight,  involving 
only  initial  training  for  the  unit  personnel.  For  the  more  complex 
elements,  the  project  surge  team  was  in  great  demand.  The  Janus  exercise 
was  anomalous:  the  project  team  prepared  the  TSP  with  full  expectation 
of  providing  a  great  deal  of  implementation  assistance.  But  the  BBSE, 
with  its  relatively  comprehensive  TSP,  also  called  for  surge  team  efforts. 

The  five  lessons  stated  below  address  three  aspects  of  supportability. 


Lesson  SI:  A  formal  program  of  education  and  orientation  should  be  implemented  to  explain  the 
FXXITP  products  to  the  installation  and  brigade  leadership,  future  leaders,  and  training 
institutions. 


Throughout  the  duration  of  the  project,  there  were  questions  about  the 
purpose  of  the  FXXITP  and  its  products.  Even  though  initial  orientation 
briefings  and  product  T3  sessions  were  conducted  at  both  user  unit 
installations,  the  chain  of  command  at  both  installations  admitted  that  they 
never  really  understood  the  products  as  well  as  they  should  have. 
Likewise,  the  NTC  personnel  had  concerns  about  the  products  that 
showed  an  incomplete  understanding  of  the  product  intent,  the  product 
purpose(s),  and  the  Army  expectations  regarding  the  use  and  support  of 
the  products  by  the  NTC  operations  group. 

In  a  separate  effort,  the  ISAT  team  conducted  a  more  comprehensive 
orientation  session  with  representatives  of  7th  ATC.  Over  a  2-day  period, 
the  ISAT  team  provided  a  hands-on  orientation  of  the  BSTS,  vignettes, 
BSE,  and  BBSE  products.  This  education  and  orientation  effort  was  felt 
to  be  very  successful,  yet  it  was  not  tied  to  a  specific  implementation 
schedule.  Rather,  the  7th  ATC  representatives  requested  the  orientation 
for  their  own  information  and  evaluation.  Perhaps  the  key  to  success  is  in 
the  motivation  of  the  recipients — whether  they  are  participating  because 
they  want  the  information,  or  because  they  are  feeling  pressured  to  add  yet 
more  briefings  to  an  already  overburdened  schedule. 


17 


Lesson  S2:  A  formal  program  of  maintenance  should  be  instituted  to  ensure  that  these  products 
remain  current  and  viable. 

Maintaining  currency  has  been  a  continuing  challenge  for  all  training 
products  being  delivered  to  the  force.  The  wider  lesson  applies  to 
products  besides  those  evaluated  in  this  project.  All  structured  training 
development  (which  is,  by  definition,  training  that  is  tied  to  doctrinal  task 
performance)  should  have  a  built-in  plan  for  maintenance.  As  described 
earlier,  maintenance  plans  for  simulation-  or  computer-based  training 
should  also  include  considerations  of  updating  the  projects  in 
synchronization  with  technological  advances. 


Lesson  S3:  A  formal  program  of  sustainment  and  support  should  be  instituted  to  ensure  that 
units  are  able  to  adapt  the  products  as  needed  and  implement  them  with  minimal  disruption  to 
unit  ongoing  requirements. 

After  concerns  about  doctrinal  currency,  the  next  most  frequent  request 
was  for  products  that  could  be  tailored  to  unit  needs  and  situations.  A 
plan  for  sustainment  of  the  products  could  include  surge  teams  that  assist 
first-time  users  or  help  with  modifications  to  products  to  meet  unit 
training  needs.  Lower  cost  off-site  assistance  could  be  offered  by  means 
of  a  hot  line  or  help  desk  and  a  web  page  for  disseminating  information. 
The  Internet-based  CITT,  described  earlier,  has  been  augmented  with  this 
type  of  direct-to-user  assistance  capability 


Lesson  S4:  If  Battle  Simulation  Center  (BSC)  contractors  are  expected  to  support  FXXITP 
product  implementation,  then  their  contracted  logistics  support  (CIS)  contracts  should  be 
written  to  include  that  requirement. 

Throughout  the  project,  there  was  a  great  deal  of  inconsistency  in  the  level 
of  support  available  at  different  simulation  centers.  One  center  said  that 
they  did  not  conduct  BBS  exercises;  another  center  said  they  were  not 
required  to  support  brigade-level  exercises  using  Janus.  This  is  contrasted 
to  the  experience  during  the  development  of  the  BSE/BBSE,  where  a  third 
center  said  they  could  support  anything  the  unit  put  in  front  of  them. 

In  order  for  structured  training  products  to  be  adequately  supported,  the 
simulation  center  CLS  contracts  should  include  language  requiring 
support  of  the  structured  training  products  as  they  were  designed.  This 
could  also  be  expanded  to  include  support  requirements  for  Training 
Support  Brigades  (TSB)  and  Divisions  and  for  the  U.S.  Army  Reserve 
Exercise  Brigades  that  could  provide  this  training  to  RC  and  AC  units  in 
the  future. 
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Lesson  S5:  The  program  and  its  products  will  require  Army-wide  support  in  order  to  be 
resourced  and  used. 


One  of  the  biggest  concerns  of  the  personnel  using  the  products  was  the 
level  of  command  support.  Without  the  proper  emphasis  from  the  Army 
chain  of  command,  most  participants  felt  that  the  products  and  the  training 
they  were  intended  to  support  would  not  be  understood,  included  in 
training  strategies  and  plans,  or  resourced. 

In  order  for  the  FXXITP  to  be  institutionalized,  support  for  the  FXXITP 
must  start  at  the  Department  of  Army  (DA)  level  and  must  be  supported 
by  each  major  command  that  will  use  the  products.  The  TRADOC  must 
not  only  present  the  program  to  personnel  attending  its  schools,  but  must 
also  be  prepared  to  maintain  the  program  and  its  products  to  ensure  their 
continued  viability  and  usefulness  to  the  units  in  the  field.  The 
FORSCOM  must  support  the  program  by  establishing  a  training  strategy 
that  helps  units  understand  how  to  incorporate  these  products  into  their 
training  plans,  and  must  also  incorporate  requirements  for  the  training  into 
FORSCOM  Regulations  350-1  and  350-2  (DA,  1998a,  1998b). 

The  products  cannot  simply  be  added  as  additional  training  requirements. 
They  must  replace  existing  but  less  efficient  or  effective  training.  One 
commander  queried  “What  can  I  take  off  my  plate  in  order  to  do  this?” 
Organizations  like  the  Combat  Training  Centers  (CTCs)  must  understand 
the  purpose  of  the  FXXITP  and  its  products  so  they  can  provide 
recommendations  to  units  on  which  FXXITP  products  they  can  train  with 
to  overcome  problems  identified  during  LTP  training  and  CTC  rotations. 

Thus,  despite  what  was  in  many  cases  a  good  understanding  of  the 
products  and  acceptance  that  the  FXXITP  products  could  help  them,  not 
all  commanders  were  able  to  use  these  products  in  lieu  of  other  training 
events  required  by  FORSCOM  and  installation  training  regulations.  This 
seems  to  have  been  primarily  because  the  program  was  a  TRADOC 
initiative  and  not  a  DA  and  FORSCOM  supported  program. 
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Training 

effectiveness 

assessment 


Lesson  TEAl: 
products. 


On  two  important  objectives  of  the  project,  we  were  totally  unsuccessful. 
We  were  unable  to  see  even  one  complete  implementation  of  any  of  the 
products,  and  we  were  unable  to  conduct  a  rigorous  assessment  of  the 
effectiveness  of  the  training  products.  As  described  earlier,  there  were  a 
number  of  reasons:  competing  unit  demands  and  missions, 
misunderstandings  about  product  utility,  lateness  of  product  introduction 
into  the  unit  training  calendars,  lack  of  understanding  of  command 
emphasis.  Despite  the  presence  of  training  coordinators  and  surge  teams, 
despite  the  orientation  sessions  and  train  the  trainer  sessions,  the  program 
was  not  implemented. 

But  even  with  a  complete  implementation,  we  would  not  have  been  able  to 
obtain  sufficient  data  to  support  a  full  training  effectiveness  assessment 
along  the  line  of  Kirkpatrick’s  model  (1994).  One  or  two  brigades,  each 
with  ongoing  uncontrolled  activities,  continuous  personnel  turnover,  and 
the  absence  of  a  control  group,  would  not  represent  acceptable  conditions 
for  such  a  study.  Additionally,  many  of  the  data  sources  were  not 
accessible:  NTC  observers  and  LTP  personnel  protect  their  clientele  and 
their  observations  closely,  quite  properly  not  permitting  any  results  to  be 
used  for  purposes  for  which  they  are  not  intended. 

This  is  not  to  say  that  the  assessment  work  yielded  no  useful  information. 
A  qualitative  assessment,  with  its  detailed  description  of  program 
implementation,  can  be  the  method  of  necessity  in  many  situations,  but  is 
often  the  method  of  choice  (Patton,  1987).  Nonetheless,  we  feel  keenly 
the  need  for  a  description  of  impact. 


We  are  still  in  need  of  a  rigorous  training  effectiveness  assessment  of  the  FXXITP 


Perhaps  the  Kirkpatrick  model  (1994)  is  not  the  correct  one  for  this  kind 
of  training  evaluation.  Quasi-experimental  methods,  such  as  are 
propounded  in  D.  T.  Campbell  and  Stanley’s  classic  work  (1966)  and 
updated  in  Cook  and  Campbell  (1979),  should  be  explored  as  more 
appropriate  possibilities  for  assessment  studies.  We  ought  not  to  abandon 
the  attempt.  The  guidance  in  a  variety  of  guides  to  program  evaluation 
(e.g..  Joint  Committee  on  Standards  for  Educational  Evaluation,  1994; 
King,  Morris,  &  Fitz-Gibbon,  1987;  Patton,  1987;  Rossi  &  Freeman, 
1993;  Shadish,  Cook,  &  Leviton,  1991)  should  be  explored  carefully  to 
determine  whether  there  are  other  approaches  that  could  be  incorporated. 
As  we  continue  to  develop  structured  training  products,  we  should  also 
continue  the  attempt  to  obtain  evidence  of  their  effectiveness. 
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Summary 


Conclusions 
and  Recommen 
dations 


Plans  for  fielding, 
maintenance,  and 
sustainment 


This  section  has  described  a  number  of  lessons  learned.  The  lessons 
address  considerations  of  acceptability,  impact,  supportability,  and 
training  effectiveness  assessment  itself.  The  lessons  are,  on  one  level, 
discouraging,  as  they  are  generally  admissions  of  shortcomings  in  the 
FXXITP  products.  Within  the  lessons,  however,  we  have  attempted  to 
offer  solutions  to  the  identified  problems.  Some  of  the  solutions  are  more 
ambitious  than  others:  Some  require  additional  research,  while  others  will 
require  action  at  the  highest  levels  of  Army  leadership.  No  one  solution 
can  solve  all  of  the  problems,  and  no  one  solution  can  be  truly  effective 
for  even  one  problem  without  implementation  of  solutions  that  address  all 
of  the  problems. 

Those  solutions  that  are  within  the  reach  of  training  designers  and 
developers  include  research  on  TSP  and  implementation  models,  redesign 
of  TSP  products  and  distribution  requirements,  and  planning  for 
maintenance  and  sustainment  of  products.  Other  solutions,  including  the 
need  for  command  emphasis  at  division-level  and  higher,  and  the 
institutionalizing  of  the  program,  will  require  that  TRADOC  and  Army 
leaders  make  a  commitment  to  support  the  development  and 
implementation  of  such  products  that  may  increase  readiness  without 
increasing  training  costs. 


This  report  has  summarized  a  set  of  13  lessons,  derived  from  observations, 
interviews,  and  analysis  of  the  questionnaire  data.  The  lessons  can  be 
further  reduced  to  three  critical  issues  that  demand  attention,  based  on  this 
research  effort: 

•  plans  for  fielding,  maintenance,  and  sustainment; 

•  methods  for  making  TSPs  more  usable;  and 

•  Army  and  TRADOC  initiatives  for  ensuring  product  use. 


Figure  1  illustrates  the  relationships  among  the  processes  of  fielding, 
maintenance,  and  sustainment.  As  shown,  initial  fielding  gradually 
transitions  to  ongoing  sustainment.  Similarly,  initial  updates  are 
succeeded  by  a  process  of  continuing  updates.  Feedback  and  lessons 
learned  during  fielding  are  used  to  make  the  initial  updates,  which  are  then 
delivered  and  incorporated  into  the  already  fielded  products.  This  process 
continues  over  time:  Units  use  the  products  and  provide  suggestions  or 
concerns,  and  developers  use  the  information  to  make  continual 
improvements.  Although  solutions  for  each  part  of  the  process  can  be 
planned  and  executed  separately,  all  three  needs  must  be  addressed  in 
order  for  any  solution  to  be  effective. 
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Figure  1.  Integration  of  fielding,  maintenance,  and  sustainment  activities  for  Force  XXI  Training 
Program  products. 


Plans  for  fielding, 
maintenance,  and 
sustainment 
(continued) 


The  plan  calls  for  a  combination  of  hot  lines  and  surge  teams  to  work  with 
first  users.  The  same  mechanisms  would  also  be  useful  for  obtaining 
feedback  from  units  using  the  materials,  for  use  in  planning  for  and 
carrying  out  the  most  urgently  needed  updates.  As  the  fielding  and 
implementation  support  are  continuing,  the  update  team  would  be 
amassing  observation,  feedback,  and  lessons  learned  concerning  the 
products  and  their  use.  This  information  would  be  used  to  determine  the 
needs  for  additional  maintenance  of  the  products. 

The  requirements  demand  an  integrated,  multifaceted  plan  that  addresses 
initial  fielding  support,  immediate  updates,  continuing  maintenance,  and 
ongoing  sustainment  for  users.  The  plan  must  take  into  consideration 
primary  users  in  the  AC  and  RC  units,  as  well  as  institutional  users  and 
supporters.  Finally,  the  plan  must  ensure  that  the  personnel  who  will 
carry  out  the  activities  are  of  the  appropriate  levels  of  expertise  and 
experience  with  respect  to  doctrine,  technology,  and  the  training  products 
themselves. 


Making  training 
products  more 
usable 


Army  emphasis  on 
product 

incorporation  in 
training  plans 


Summary 


The  objective  is  worthy:  TSP  materials  should  be  extremely  user-friendly 
and  should  present  no  barriers  to  use.  For  years,  training  researchers  have 
been  working  on  visions  of  materials  that  participants  would  use  as 
designed.  Despite  design  innovations  and  increasingly  simplified  and 
comprehensive  presentations,  however,  we  find  repeatedly  that  guides  and 
instructions  are  not  used.  Directed  research  on  what  works,  for  what 
products,  under  what  conditions,  would  be  useful  in  addressing  this 
objective.  But  the  FXXITP  and  the  Army  should  also  be  prepared  to 
insist  that  users  attempt  to  use  the  products.  Until  we  get  a  full  effort  at 
implementation,  we  are  still  guessing  at  optimal  formats  and  models. 


All  we  can  do  with  this  recommendation  is  to  ask  decision-makers  to 
make  their  decision:  Do  they  want  the  products  to  be  used?  If  so,  then 
they  must  provide  the  resources — time  and  personnel — and  the  edict  that 
tells  units  that  the  products  are  to  be  used.  They  must  invest  in  a  fielding, 
maintenance,  and  sustainment  plan  that  will  keep  the  products  viable. 

As  institutionalized  training  programs,  the  LTP,  CTC  rotations.  Battle 
Command  Battle  Staff  Training  for  RC  units,  and  other  such  events 
“work.”  Why  is  that?  These  training  programs  have  an  established 
infrastructure  that  includes  expert  personnel  and  well-tested  training 
approaches  to  ensure  support  for  user  units.  They  are  a  part  of  the  system: 
Units  know  that  they  must  participate,  and  are  eager  to  do  so.  The 
programs’  reputation  for  effective  rotation  logistics  and  tough,  thorough 
training  ensure  units  that  their  time  will  be  well  spent.  Since  the  CTCs 
and  other  similar  programs  began  to  be  a  part  of  the  Army  training  system 
(in  the  broadest  sense),  they  have  become  ingrained  in  the  training  culture. 

This  should  be  the  vision  for  the  FXXITP.  It  will  take  time,  foresight,  and 
commitment  of  resources  and  planning.  Insights  drawn  from  this  project 
should  be  useful  in  thinking  through  the  infrastructure  and  education 
needs.  Lessons  learned  from  the  earliest  days  of  CTC  history  would  also 
inform  the  planning  process.  Finally,  research  from  organizations  such  as 
ARI  could  continue  to  guide  the  institutionalization  of  the  program. 


The  ISAT  project  has  provided  a  wealth  of  valuable  information  to  the 
FXXITP  and  the  larger  training  community.  The  quantitative  data  were 
sparse  compared  to  the  richness  of  the  observations  and  interviews.  But  in 
training  effectiveness  assessments,  even  when  quantitative  data  are 
available,  the  analyses  and  results  must  be  supplemented  with  comments 
and  observations.  Thus  any  disappointment  with  the  amount  of  data  is 
more  than  relieved  by  satisfaction  with  the  qualitative  information,  the 
honest  and  constructive  input  from  users,  and  the  insights  provided  by 
reviewers.  The  recommendations  and  lessons  learned  should  assist  the 
FXXITP,  TRADOC,  and  the  Army  to  continue  their  attempts  to  provide 
training  that  improves  readiness  and  is  accessible  by  units. 
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AFRU 

ARI 
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BSTS 

CBI 

cm 

CLS 

COBRAS 
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CSS 
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FTX 

FXXITP 

HumRRO 

IFRU 
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Appendix  A 
Acronym  List 


after  action  review 

active  component 

Armored  Forces  Research  Unit 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Army  Training  Center 

Brigade/Battalion  Battle  Simulation 
Brigade  and  Battalion  Staff  Exercise 
Battle  Simulation  Center 
Brigade  Staff  Exercise 
Battle  Staff  Training  System 

computer-based  instruction 
Commanders’  Integrated  Training  Tool 
contracted  logistics  support 

Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved  Through 
Simulation 
command  post 
combat  support 
combat  service  support 
Combat  Training  Center 

Department  of  the  Army 

Deputy  Chief  of  Staff  for  Training 

Directorate  of  Training  and  Doctrine  Development 

Environment  for  MultiMedia  interactive  instruction 

U.S.  Forces  Command 
field  training  exercise 
Force  XXI  Training  Program 

Human  Resources  Research  Organization 

Infantry  Forces  Research  Unit 

Implementation  and  Support  for  the  Assessment  of  Force  XXI  Training 
Program 

local  area  network 
Leader  Training  Program 


A-l 


NCO 

NTC 

noncommissioned  officer 

National  Training  Center 

OPFOR 

opposing  force 

RC 

reserve  component 

SGE 

SME 

SOP 

Small  Group  Exercises 
subject  matter  expert 
standing  operating  procedure 

T3 

TMS 

TRADOC 

TSB 

TSC 

TSP 

train  the  trainer 

training  management  system 

U.S.  Army  Training  and  Doctrine  Command 
Training  Support  Brigade 

Training  Support  Coordinator 
training  support  package 

USAARMC 

U.S.  Army  Armor  Center 

